Tools for modelling and simulating migration-based preservation

This report describes two tools for modelling and simulating the costs and risks of using IT storage systems for the long-term archiving of file-based AV assets. The tools include a model of storage costs, the ingest and access of files, the possibility of data corruption and loss from a range of mechanisms, and the impact of having limited resources with which to fulfill access requests and preservation actions. Applications include archive planning, development of a technology strategy, cost estimation for business planning, operational decision support, staff training and generally promoting awareness of the issues and challenges archives face in digital preservation

Self-organising agent communities for autonomic computing

Efficient resource management is one of key problems associated with large-scale distributed computational systems. Taking into account their increasing complexity, inherent distribution and dynamism, such systems are required to adjust and adapt resources market that is offered by them at run-time and with minimal cost. However, as observed by major IT vendors such as IBM, SUN or HP, the very nature of such systems prevents any reliable and efficient control over their functioning through human administration.For this reason, autonomic system architectures capable of regulating their own functioning are suggested as the alternative solution to looming software complexity crisis. Here, large-scale infrastructures are assumed to comprise myriads of autonomic elements, each acting, learning or evolving separately in response to interactions in their local environments. The self-regulation of the whole system, in turn, becomes a product of local adaptations and interactions between system elements.Although many researchers suggest the application of multi-agent systems that are suitable for realising this vision, not much is known about regulatory mechanisms that are capable to achieve efficient organisation within a system comprising a population of locally and autonomously interacting agents. To address this problem, the aim of the work presented in this thesis was to understand how global system control can emerge out of such local interactions of individual system elements and to develop decentralised decision control mechanisms that are capable to employ this bottom-up self-organisation in order to preserve efficient resource management in dynamic and unpredictable system functioning conditions. To do so, we have identified the study of complex natural systems and their self-organising properties as an area of research that may deliver novel control solutions within the context of autonomic computing.In such a setting, a central challenge for the construction of distributed computational systems was to develop an engineering methodology that can exploit self-organising principles observed in natural systems. This, in particular, required to identify conditions and local mechanisms that give rise to useful self-organisation of interacting elements into structures that support required system functionality. To achieve this, we proposed an autonomic system model exploiting self-organising algorithms and its thermodynamic interpretation, providing a general understanding of self-organising processes that need to be taken into account within artificial systems exploiting self-organisation.<br/

Self-organising agent communities for autonomic resource management

The autonomic computing paradigm addresses the operational challenges presented by increasingly complex software systems by proposing that they be composed of many autonomous components, each responsible for the run-time reconfiguration of its own dedicated hardware and software components. Consequently, regulation of the whole software system becomes an emergent property of local adaptation and learning carried out by these autonomous system elements. Designing appropriate local adaptation policies for the components of such systems remains a major challenge. This is particularly true where the system’s scale and dynamism compromise the efficiency of a central executive and/or prevent components from pooling information to achieve a shared, accurate evidence base for their negotiations and decisions.In this paper, we investigate how a self-regulatory system response may arise spontaneously from local interactions between autonomic system elements tasked with adaptively consuming/providing computational resources or services when the demand for such resources is continually changing. We demonstrate that system performance is not maximised when all system components are able to freely share information with one another. Rather, maximum efficiency is achieved when individual components have only limited knowledge of their peers. Under these conditions, the system self-organises into appropriate community structures. By maintaining information flow at the level of communities, the system is able to remain stable enough to efficiently satisfy service demand in resource-limited environments, and thus minimise any unnecessary reconfiguration whilst remaining sufficiently adaptive to be able to reconfigure when service demand changes

Energy, entropy and work in computational ecosystems: a thermodynamic account

Recently, computer scientists have begun to build computational ecosystems in which multiple autonomous agents interact locally to achieve globally efficient organised behaviour. Here we present a thermodynamic interpretation of these systems. We highlight the difference between the regular use of terms such as energy and work, and their use within a thermodynamic framework. We explore the way in which this perspective might influence the design and management of such systems

Emergent Service Provisioning and Demand Estimation through Self-Organizing Agent Communities

A major challenge within open markets is the ability to satisfy service demand with an adequate supply of service providers, especially when such demand may be volatile due to changing requirements, or fluctuations in the availability of services. Ideally, the supply and demand of services should be balanced; however, when consumer demand change over time, and providers can independently choose which services they provide, a coordination problem known as ‘herding’ can arise, bringing instability to the market. This behavior can emerge when consumers share similar preferences for the same providers, and thus compete for the same resources. Likewise, providers which share estimates of fluctuating service demand may respond in unison, withdrawing some services to introduce others, and thus oscillate the available supply around some ideal equilibrium. One approach to avoid this unstable behavior is to limit the flow of information across the agent community, such that agents possess an incomplete and subjective view of the local service availability and demand. By drawing inspiration from information flow within biological systems, we propose a model of an adaptive service-offering mechanism, in which providers adapt their choice of services that they offer to consumers, based on perceived demand. By varying the volume of information shared by agents, we demonstrate that a co-adaptive equilibrium can be achieved, thus avoiding the herding problem. As the knowledge that agents can possess is limited, agents self-organise into community structures that support locally shared information. We demonstrate that such a model is capable of reducing instability in service demand and thus increase utility (based on successful service provision) by up to 59%, when compared to the use of globally available information

Tools for quantitative comparison of preservation strategies

This report describes the tools developed by IT Innovation for quantitative comparison of preservation strategies. The tools have been open sourced and are publicly available on a website which includes documentation and features for bug reporting and new functionality requests

Self-organising agent communities for autonomic computing

Efficient resource management is one of key problems associated with large-scale distributed computational systems. Taking into account their increasing complexity, inherent distribution and dynamism, such systems are required to adjust and adapt resources market that is offered by them at run-time and with minimal cost. However, as observed by major IT vendors such as IBM, SUN or HP, the very nature of such systems prevents any reliable and efficient control over their functioning through human administration. For this reason, autonomic system architectures capable of regulating their own functioning are suggested as the alternative solution to looming software complexity crisis. Here, large-scale infrastructures are assumed to comprise myriads of autonomic elements, each acting, learning or evolving separately in response to interactions in their local environments. The self-regulation of the whole system, in turn, becomes a product of local adaptations and interactions between system elements. Although many researchers suggest the application of multi-agent systems that are suitable for realising this vision, not much is known about regulatory mechanisms that are capable to achieve efficient organisation within a system comprising a population of locally and autonomously interacting agents. To address this problem, the aim of the work presented in this thesis was to understand how global system control can emerge out of such local interactions of individual system elements and to develop decentralised decision control mechanisms that are capable to employ this bottom-up self-organisation in order to preserve efficient resource management in dynamic and unpredictable system functioning conditions. To do so, we have identified the study of complex natural systems and their self-organising properties as an area of research that may deliver novel control solutions within the context of autonomic computing. In such a setting, a central challenge for the construction of distributed computational systems was to develop an engineering methodology that can exploit self-organising principles observed in natural systems. This, in particular, required to identify conditions and local mechanisms that give rise to useful self-organisation of interacting elements into structures that support required system functionality. To achieve this, we proposed an autonomic system model exploiting self-organising algorithms and its thermodynamic interpretation, providing a general understanding of self-organising processes that need to be taken into account within artificial systems exploiting self-organisation.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Planning and managing the 'cost of compromise' for AV retention and access

Long term retention and access to AV assets as part of a preservation strategy inevitably involves some form of compromise in order to achieve acceptable levels of cost, throughput, quality and many other parameters. Examples include: quality control and throughput in media transfer chains; data safety and accessibility in digital storage systems; and service levels for ingest and access for archive functions delivered as services. We present new software tools and frameworks developed in the PrestoPRIME project that allow these compromises to be quantitatively assessed, planned and managed for file-based AV assets. Our focus is how to give an archive an assurance that when they design and operate a preservation strategy as a set of services that it will function as expected and can cope with the inevitable and often unpredictable variations that will happen in operation. This includes being able to do cost projections, sensitivity analysis, simulation of 'disaster scenarios’, and to govern preservation services using Service Level Agreements and policies